Add limit to total number of fields in mapping #17357
Conversation
This is to prevent mapping explosion when dynamic keys such as UUID are used as field names. index.mapping.total_fields.limit specifies the total number of fields an index can have. An exception will be thrown when the limit is reached. The default limit is 1000. Value 0 means no limit. This setting is runtime adjustable
|
I wonder if we really need to make |
s1monw
added
>enhancement
review
:Search/Mapping
| @@ -403,6 +406,13 @@ private void checkNestedFieldsLimit(Map<String, ObjectMapper> fullPathObjectMapp | |||
| } | |||
| } | |||
|
|
|||
| private void checkTotalFieldsLimit(long totalMappers) { | |||
| long allowedTotalFields = indexSettings.getValue(INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING); | |||
| if (allowedTotalFields > 0 && allowedTotalFields < totalMappers) { | |||
There was a problem hiding this comment.
Like Simon suggested, I would just do: if (allowedTotalFields < totalMappers) {. If someone needs to have many fields anyway and understands the issue, (s)he can still set index.mapping.total_fields.limit=1000000 for instance.
|
@simonw The code has the check under the following |
|
@jpountz good can we add a comment there? |
|
I will do it in a separate commit. |
|
thx @jpountz |
This is to prevent mapping explosion when dynamic keys such as UUID are used as field names. index.mapping.total_fields.limit specifies the total number of fields an index can have. An exception will be thrown when the limit is reached. The default limit is 1000. Value 0 means no limit. This setting is runtime adjustable Closes #11443
|
@jpountz Not at all. Thanks for merging this feature. |
…onIT#testDynamicUpdates With restriction for the total number of fields introduced in #17357 this test can fail if a large number of records is randomly selected for indexing.
|
@jpountz is there a chance that this PR can get back ported to 2.3.x ? I've discussed this with @yanjunh who pointed me to a fork 2.3.1 with this fix here: https://github.com/yanjunh/elasticsearch/tree/v2.3.1-maplimit |
|
@segalziv no, this is a breaking change so we'll keep it in 5.0 only |
|
ok @clintongormley , thanks anyway |
|
@yanjunh Is this limitation is a hard value for ES usage ? because the setting will let us think it a red line, and in current USE CASE ,we have documents exceed 3000 fields, so we have to divide the document to different index? is there any bad things to exceed 10000 fields? |
|
@thundercrawl The default limit is rather conservative. Depending on the resource (especially memory) ES nodes have, you can config much large limit. In practice, I have seen people using 10000 without problem. |
|
@yanjunh Great thanks, that helpful for us to make a design |
This is to prevent mapping explosion when dynamic keys such as UUID are used as field names. index.mapping.total_fields.limit specifies the total number of fields an index can have. An exception will be thrown when the limit is reached. The default limit is 1000. Value 0 means no limit. This setting is runtime adjustable